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United  States 

General  Accounting  OfBce 

Washington,  D.C.  20548 


Resources,  Community,  and 
Economic  Development  Division 

B-270651 

Januaiy  11, 1996 

The  Honorable  George  E.  Brown,  Jr. 
Ranking  Minority  Member 
Committee  on  Science 
House  of  Representatives 

Dear  Mr.  Brown: 


This  report  responds  to  your  request  that  we  assess  the  impact  of  the 
Advanced  Technology  Program  (atp),  which  is  administered  by  the 
National  Institute  of  Standards  and  Technology  within  the  Department  of 
Commerce,  atp’s  puipose  is  to  provide  support  on  a  cost-sharing  basis  for 
indv^trial  research  and  development  projects — ^projects  that  have  a 
signific^t  potential  for  stimulating  economic  growth  and  improving  the 
competitiveness  of  U.S.  industry.  Federal  funding  for  atp  has  grown 
sharply,  from  $68  million  in  fiscal  year  1993  to  $341  million  in  fiscal  year 
1996.  Recently,  however,  budget  proposals  have  suggested  eliminating 
ATP  s  funding  for  fiscal  year  1996. 

As  agreed  with  your  office,  our  objective  was  to  examine,  as  one  way  to 
^sess  atp’s  impact,  whether  research  projects  would  have  been  funded  by 
the  private  sector  if  they  had  not  received  funds  from  atp.  We  also 
examined  atp’s  impact  in  terms  of  other  goals  of  the  program,  such  as 
aiding  the  formation  of  joint  ventures.  We  agreed  on  this  approach 
because  of  the  difficulty  of  assessing  the  net  impact  of  atp’s  investments  in 
technology  on  the  economy.  For  example,  it  is  difficult  to  establish  a 
causal  link  between  a  successful  project  and  government  funding  earlier  in 
the  project.  Moreover,  the  impact  of  atp  should  be  measured  not  only  by 
rts  effect  on  the  firms  that  receive  funding  but  also  by  its  effect  on  other 
firms— a  difficult  undertaking  that  our  approach  avoids. 

To  meet  our  objective,  we  focused  on  two  groups  of  atp  applicants,  which 
we  called  winners”  and  “near  winners.”  Both  groups  submitted  proposals 
that  were  rated  highest  during  atp’s  review,  but  the  near  winners  did  not 
idtimately  receive  atp  funding.  We  surveyed  by  telephone  all  applicants 
that  qualified  as  wiimers  or  near  wirmers  during  atp’s  first  4  years 
(1990-93).  We  achieved  a  100-percent  response  rate  from  the  123 
respondents  that  we  included  in  our  analysis  (89  winners  and  34  near 
winners).  We  asked  the  near  winners  if  they  had  continued  their  proposed 
projects  usmg  other  funding  sources  after  atp  declined  to  fund  them. 

Given  the  similarity  in  the  qualifications  of  the  proposals  submitted  by  the 


Ik 


Page  1 


GAO/RCED-96-47  Advanced  Technology  Program 


B-270651 


winners  and  near  winners  as  determined  by  ATP,  another  purpose  of  this 
question  was  to  determine  whether  the  winners  would  have  been  likely  to 
continue  their  projects  without  atp  funding.  We  also  asked  both  groups  if 
they  had  sought  funding  from  other  sources  before  applying  to  atp.  This 
question  provided  information  on  whether  private-sector  sources  had  the 
opportunity  to  fund  proposed  projects  before  an  applicant  sought  atp 
funding. 

In  our  survey,  we  also  collected  information  that  provides  an  extensive 
profile  of  the  respondents,  which  we  used  in  some  of  our  analyses; 
additional  information  from  this  profile  is  provided  in  appendix  I. 
Appendix  n  contains  our  survey  questions  and  an  aggregate  list  of  all  the 
responses,  and  appendix  in  provides  further  detail  about  our  objectives, 
scope,  and  methodology. 


atp  has  funded  research  projects  that  would  have  been  funded  by  the 
R0SU.ltS  in  Jl5lT0l  private  sector  as  well  as  those  that  would  not.  Half  of  the  near  winners 

continued  their  projects  without  relying  on  atp  funding,  wMle  the  other 
half  discontinued  their  projects  for  various  reasons.  The  winners  were 
nearly  evenly  divided  when  asked  if  they  would  have  pursued  their 
projects  even  if  they  had  not  received  atp  funding.  Almost  all  the  near 
winners  that  continued  their  projects  did  so  on  a  modified  schedule, 
meeting  the  projects’  milestones  later  than  scheduled  in  their  proposals  to 

ATP. 

In  most  cases,  private-sector  sources  did  not  have  the  opportumty  to  fund 
ATP  projects.  Of  the  123  applicants  we  surveyed,  77,^  or  63  percent,  did  not 
look  for  funding  from  other  sources  before  requesting  it  from  atp.  Those 
applicants  that  did  look  for  funding  looked  for  a  long  time  and  made  many 
attempts  to  find  funding,  on  average.  Seven  applicants  turned  down  offers 
from  private  sources  because  they  could  not  reach  an  acceptable  funding 
arrangement. 

Our  survey  also  found  that  atp  had  other  effects.  More  than  three-fourths 
of  the  joint-venture  applicants  indicated  that  they  had  come  together 
solely  to  pursue  an  atp  project,  thus  satisfying  atp’s  goal  of  serving  as  a 
catalyst  for  the  formation  of  joint  ventures.  Furthermore,  of  the  46 
applicants  that  tried  to  find  funding  elsewhere  before  turning  to  atp,  about 
half  were  told  by  prospective  funders  that  their  projects  were  either  too 


^One  applicant  did  not  know. 
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risky  or  “precompetitive”^ — characteristics  that  fulfill  the  aims  of  atp 
funding. 


Background  mission  is  to  stimulate  economic  growth  in  the  United  States  through 

technology  development.  The  program  seeks  to  accomplish  that  mission 
by  sharing  the  cost  of  industrial  research  and  development  projects.  The 
projects  selected  by  atp  for  funding  are  characterized  by  “a  potential 
broad-based  economic  impact  but  a  relatively  high  technical  risk  and  a 
long  time  horizon,”  according  to  atp. 

ATP  s  guidance  states  that  if  the  technical  risk  associated  with  a  project  is 
very  low,  federal  funding  should  not  be  necessary.  In  addition,  when 
submitting  a  research  proposal,  applicants  mxrst  sign  a  form  stating  that 
this  proposal  is  not  requesting  funding  for  existing  or  planned  research 
programs  that  would  be  conducted  in  the  same  time  period  in  the  absence 
of  financial  assistance  under  the  atp.”  This  wording  suggests  that  atp 
should  not  fimd  projects  that  other  sources  would  have  funded  or,  when 
ATP  does  fund  such  projects,  that  atp  funds  should  enable  applicants  to 
complete  their  projects  in  a  shorter  time. 


Applicants’  Actions 
and  Intentions  to  Find 
Funding  Before 
Applying  to  ATP 


Most  ATP  applicants  did  not  look  for  funding  from  other  sources  before 
requesting  it  fi'om  atp.  In  addition,  the  applicants  were  about  evenly 
divided  when  asked  if  they  intended  to  pursue  their  projects  whether  or 
not  they  received  atp  funding. 

When  asked  if  they  had  searched  for  funding  from  other  sources  before 
applying  to  atp,  63  percent  of  the  applicants  (77  of  123;  one  appUcant  did 
not  know)  said  that  they  had  not.  Considering  the  winners  and  near 
winners  separately,  we  found  that  65  percent  (58  of  89)  of  the  winners  had 
not  looked  for  funding  before  applying  to  atp,  along  with  56  percent  (19  of 
34)  of  the  near  winners. 


Of  the  45  applicants  that  had  looked  for  funding  before  applying  to  atp 
about  53  percent  (24  of  45)  sought  it  fi-om  private  sources  only,  9  percent 
(4)  fi-om  public  sources  only,  and  38  percent  (17)  fi-om  some  combination 
of  pnvate  and  pubUc  sources.  On  average,  42  of  these  45  respondents  (3 
did  not  respond)  searched  for  funding  for  18  months  before  applying  to 
ATP  and  made  eight  separate  attempts  to  find  funding. 


research  and  development  at  which  a  preliminaiy 
^Xped  “  ®  commercial  potential  can  be  made  but  before  commerc^lStypes 


are 
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When  asked  what  reasons  the  non-ATP  sources  gave  for  not  providing 
funding,  54  percent  of  these  applicants  (22  of  41;  4  did  not  respond)  said 
that  their  projects  were  viewed  as  either  too  risky  or 

“precompetitive”— both  outlined  in  Conunerce’s  regulations  as  reasons  for 
ATP  to  fund  projects.  Sixteen  percent  (7  of  45)  said  they  had  turned  down 
funding  offers  because  they  could  not  agree  on  terms  with  the  prospective 
funder;  3  of  these  7  eventually  received  atp  funding. 

We  asked  the  winners  and  near  winners  if  they  intended  to  pursue  their 
projects  whether  or  not  they  received  atp  funding.  When  we  considered 
the  two  groups  together,  42  percent  (52  of  123)  said  “yes”  or  “probably 
yes;”  41  percent  (51)  said  “no”  or  “probably  no;”  and  16  percent  (20)  were 
uncertain.  Of  the  respondents  that  said  they  intended  to  pursue  the 
project,  94  percent  (49  of  52)  indicated  that  their  projects’  schedules 
would  be  modified  and  that  the  milestones  would  be  met  later  than 
scheduled  in  their  proposals  to  atp.  When  we  considered  the  atp  winners’ 
answers  alone,  40  percent  (36  of  89)  said  “yes  or  probably  yes, 

16  percent  (14)  were  undecided;  and  44  percent  (39)  said  no  or  probably 

no.” 

Most  of  the  joint-venture  applicants  came  together  to  apply  to  atp. 
Seventy-six  percent  (26  of  34)  said  they  had  formed  a  new  group  to  pursue 
the  projects  described  in  their  atp  proposals.  'The  remaining  joint-venture 
applicants  had  worked  together  before  applying  to  atp,  pursuing  either  the 
projects  that  they  proposed  to  atp  or  other  projects. 


Near  Winners’  Actions 
After  ATP  Declined  to 
Fund  Their  Proposals 


After  atp  declined  to  fund  their  proposals,  half  of  the  near  winnere 
continued  their  projects  using  other  funding  sources.  'The  near  winners 
with  certain  characteristics  were  more  likely  to  continue  their  projects 
than  others.^ 


Half  of  Near  Winners 
Continued  Projects  Using 
Other  Funding  Sources 


Half  of  the  near  winners  (14  of  28)  continued  their  projects  using  other 
funding  sources  after  atp  declined  to  fund  them.  These  other  funding 
sources  included  federal  government  programs  other  than  atp;  state 
government  agencies;  and  private  funders,  such  as  industry  groups  or 
trade  associations,  other  private  companies,  venture  capitalists,  and  the 
company  itself.  All  14  near  winners  used  some  private  funding  to  continue 


3ln  our  findings  for  this  section  only,  the  total  of  near  winners  drops  to  28  because  6  of  the  near 
winners  were  granted  funding  by  ATP  in  a  subsequent  round  of  competition.  Thus,  we  eliraina^  them 
from  consideration  here,  focusing  only  on  those  that  found  funding  from  sources  other  than  ATP. 
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their  projects;  8  of  these  financed  their  projects  using  private  funds  only.^ 
Ninety-three  percent  (13  of  14)  of  the  projects  that  were  continued  were  or 
are  being  carried  out  on  a  modified  schedule,  meeting  their  milestones 
later  than  scheduled  in  the  proposals  submitted  to  atp. 


Some  Groups  of  Applicants 
Were  More  Likely  Than 
Others  to  Continue  Their 
Projects  Using  Other 
Funding  Sources 


Some  near  winners  were  more  likely  than  others  to  continue  their  projects 
after  atp  declined  to  fund  them.  For  example,  86  percent  (12  of  14)  of  the 
near  winners  whose  projects  were  under  way  before  they  applied  to  atp 
contoued  them,  compared  with  14  percent  (2  of  14)  of  those  whose 
projects  were  not  under  way.  Similarly,  77  percent  (10  of  13)  of  the  near 
winners  that  had  looked  for  flmding  from  other  sources  before  applying  to 
ATP  continued,  compared  with  27  percent  (4  of  15)  that  had  not  looked  for 
fimding  before  applying.  Table  1  groups  the  near  winners  according  to 
different  characteristics  and  shows  odds  ratios,  which  indicate  the  degree 
of  association  between  the  characteristics  of  these  groups  and  the 
likelihood  of  their  continuing  their  projects.  Odds  ratios  measure  the 
association  between  two  variables  through  a  single  value.  The  closer  the 
odds  ratio  is  to  1.00,  the  weaker  the  association.  For  more  information  on 
odds  ratios,  see  appendix  TTT 


Some  of  Ihe  nearji^ers  that  continued  their  projects  using  other  funding  sources  likely  benefited 
other  fun^,  from  having  been  rated  highly  by  ATP.  We  did  not  evaluate  the 

halo  effect  may  have  occurred;  however,  we  acknowledge  that  a  high  rating 
from  ATP  imght  have  proved  beneficial  to  some  near  winners  ^  ^ 
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Table  1:  Odds  Ratios  Showing 
Association  Between  Characteristics 
of  Certain  Near  Winners  and 
Continuation  of  Their  Projects 


Characteristics  of  near 
winner  A 

Characteristics  of  near 
winner  B 

Odds  ratio:  How  many 
times  more  likely  was 
near  winner  A  to  continue 
the  project  than  near 
winner  B? 

Project  under  way  before 
near  winner  applied  to  ATP 

Project  not  under  way 
before  near  winner  applied 
to  ATP 

36 

Looked  for  funding  from  other 
sources  before  applying  to 
ATP 

Did  not  look  for  funding 
from  other  sources  before 
applying  to  ATP 

9.17 

Funds  50  percent  or  more  of 
research  and  development 
internally 

Funds  less  than  50  percent 
of  research  and 
development  internally 

2.5^ 

Single  applicant 

Joint  venture 

2.75 

Company  with  more  than  10 
FTEs“ 

Company  with  10  or  fewer 
FTEs 

1.6^ 

Company  with  more  than  50 
FTEs 

Company  with  50  or  fewer 
FTEs 

1.6 

Company  with  more  than  100 
FTEs 

Company  with  100  or  fewer 
FTEs 

Company  with  more  than  500 
FTEs 

Company  with  500  or  fewer 
FTEs 

1.05 

“Nine  companies  did  not  indicate  what  percentage  of  their  research  and  development  they  fund 
internally;  therefore,  they  are  not  included  in  this  calculation. 


^RE  =  full-time  equivalent. 

°Nine  companies  did  not  provide  their  number  of  REs;  therefore,  they  are  not  included  in  this 
calculation. 

“Because  this'odds  ratio  is  close  to  1.00,  we  can  say  that  the  odds  of  a  company  with  100  or 
more  FTEs  continuing  its  project  are  about  the  same  as  the  odds  of  one  with  fewer  than  100  FTEs 
continuing  its  project. 


In  addition,  single  applicants  more  often  continued  their  projecte  than 
joint-venture  applicants:  68  percent  (11  of  19)  of  the  single  applicants 
continued  their  projects,  while  only  33  percent  (3  of  9)  of  the  joint-venture 
applicants  continued  theirs.  Seventy-one  percent  (5  of  7)  of  the  companies 
that  generally  fund  50  percent  or  more  of  their  research  and  development 
from  their  own  internal  company  funds  continued  their  projects.  In 
contrast,  50  percent  (6  of  12)  of  the  companies  that  generally  fund  less 
than  50  percent  of  their  research  and  development  from  their  own  internal 
company  funds  continued  their  projects.  (Nine  companies  did  not  indicate 
what  percentage  of  their  research  and  development  budgets  they  fund 
internally.) 
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Fin^y,  among  the  near  winners,  the  smaller  companies  continued  their 
projects  somewhat  less  frequently  than  the  larger  companies.®  For 
example,  50  percent  (3  of  6)  of  the  near-winner  companies  with  50  or 
fewer  full-time  equivalents  (fte)®  continued  their  projects,  while 
62  percent  (8  of  13)  of  the  near-winner  companies  with  more  than  50  ftes 
continued  theirs.  (Nine  compames  did  not  provide  the  number  of  ftes  for 
their  companies.) 


Fourteen  near  winners  did  not  continue  their  projects  after  atp  declined  to 
fund  the  projects.  The  reason  they  most  often  gave  for  not  continuing  was 
lack  of  funding  (cited  by  11  near  winners).  Two  near  winners  indicated 
that  their  projects  were  too  long-term;  one  cited  market  changes;  one  said 
that  the  project  was  too  risky;  and  one  joint-venture  near  winner  said  that 
its  newly  formed  partnership  had  not  worked  out.  (The  near  winners  could 
provide  more  than  one  reason.) 

Of  the  near  winners  that  did  not  continue  their  projects,  64  percent  (9  of 
14)  searched  for  funding  but  did  not  find  it.  Eight  of  these  nine  reapplied 
for  ATP  funding  during  a  subsequent  round  of  competition.  Six  of  the  nine 
are  no  longer  looking  for  funding  to  continue  their  projects. 


ATP  funded  89  projects  from  1990  to  1993,  and  14  near  winners  carried  out 
their  projects  using  other  funding  sources  during  this  time.  Sixty-four 
percent  of  these  projects  (66  of  103)  were  still  under  way  at  the  time  of  our 
survey.  Twenty-seven  percent  of  the  projects  (28  of  103)  had  been 
completed,  while  5  had  been  discontinued.  The  respondents  to  our  survey 
listed  the  status  of  the  four  remaining  projects  as  either  suspended  or 
delayed.  When  we  asked  all  the  applicants  that  had  carried  out  then- 
projects  how  satisfied  they  were  with  either  the  projects’  technical 
direction  and  progress  (for  ongoing  projects)  or  outcome  (for  completed 
projects),  94  percent  (84  of  89)  of  the  winners  and  79  percent  (11  of  14)  of 
the  near  winners  whose  projects  were  funded  by  other  sources  indicated 
that  they  were  at  least  generally  satisfied. 


^"ere^^i^s  included  in  this  comparison  because  they  may  include  companies  of 

'Measures  of  FTE  indicate  a  company’s  size  by  estimating  how  many  full-time  employees  are 
represented  by  all  part-time  and  full-time  employees  considered  togetlier. 


Status  of  Projects  Funded 
by  ATP  and  Other  Sources 


Some  Near  Winners  Did 
Not  Continue  Their 
Projects 
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According  to  the  results  of  our  survey,  atp  funds  both  projects  that  would 
Conclusions  funded  in  its  absence  and  projects  that  would  not  have  been 

funded.  In  addition,  atp  achieves  other  goals,  such  as  aiding  the  formation 
of  joint  ventures  and  helping  companies  achieve  research  milestones 
faster.  These  results  should  be  considered  together  when  assessing  atp’s 
impact. 


We  provided  a  draft  of  this  report  to  the  Department  of  Commerce  for 
Agency  Comments  comment,  commerce  found  much  of  the  report  to  be  well  done  but 

and  Our  Evaluation  recommended  certain  changes.  For  example.  Commerce  felt  that  the 

report  needed  to  highlight  our  survey’s  results  showing  that  those  research 
projects  of  near  winners  that  were  continued  with  alternative  funding 
continued  at  a  slower  pace  than  planned.  We  have  revised  the  report  as 
appropriate.  Commerce’s  written  comments,  along  with  our  detailed 
responses,  are  provided  in  appendix  IV. 


We  conducted  our  assessment  from  July  1994  through  December  1995  in 
accordance  with  generally  accepted  government  auditing  standards.  As 
arranged  with  yom  office,  unless  you  publicly  announce  its  contents 
earlier,  we  plan  no  further  distribution  of  this  report  until  30  days  after  the 
date  of  this  letter.  At  that  time,  we  wiU  send  copies  to  the  Secretary  of 
Commerce;  the  Director,  National  Institute  of  Standards  and  Technology; 
the  Director,  atp;  the  Inspector  General,  Department  of  Commerce;  the 
Director,  Office  of  Management  and  Budget;  and  other  interested  parties. 
We  will  also  make  copies  available  to  others  on  request. 

Please  call  me  at  (202)  612-3841  if  you  or  your  staff  have  any  questions. 
Mqjor  contributors  to  this  report  are  listed  in  appendix  V. 

Sincerely  yours, 


Victor  S.  Rezendes 
Director,  Energy  and 
Science  Issues 
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Appendix  I _ _ _ _ _ 

Profile  Information  for  Single-Applicant 
Companies 


In  our  survey,  we  asked  individual  applicants  for  general  information 
which,  taken  together,  provides  a  profile  of  their  companies.  We  used 
some  of  this  information  for  the  analysis  summarized  in  table  1  of  this 
report.  We  include  here  supplemental  profile  information  that  provides  an 
overall  picture  of  the  coverage  provided  by  the  Advanced  Technology 
Program  (atp).  This  information  includes  the  size  of  the  compames  based 
on  the  number  of  employees  as  well  as  on  gross  sales.  We  requested  these 
figures  for  the  fiscal  year  completed  before  the  company  applied  to  atp. 

We  also  asked  for  the  year  the  company  made  its  first  sale  as  an  indicator 
of  the  age  of  the  applicant  company.  Included  also  are  figures  for  the 
sources  that  each  company  relies  on  for  its  overall  research  budget.  We 
again  asked  companies,  in  answering  this  question,  to  base  their  responses 
on  the  fiscal  year  completed  before  they  applied  to  atp. 


Figure  1,1 :  Number  of  Full-Time 

Equivalent  Employees,  Including  Both  so  Number  of  atp  Applicants 

Outsourced  and  Permanent 

Employees 

40 


30 


Full-Time  Equivalents  (FTEs) 
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Profile  Information  for  Single-Applicant 
Companies 


Figure  1.2:  Total  Value  of  Gross  Sales 
in  Fiscal  Year  Before  Company’s 
Application  to  ATP 
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Profile  Information  for  Single-Applicant 
Companies 


Figure  1.3:  Year  of  Company’s  First  Sales 
Number  of  ATP  Applicants 


Year 
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Profile  Information  for  Sin^e-Applicant 
Companies 


Figure  1.4:  Percentage  of  Company’s 
Direct  Research  and  Deveiopment 
(R&D)  Budget  Provided  by  Federai 
Government  Agencies 


100  Number  of  ATP  Applicants 
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Figure  1.5:  Percentage  of  Company’s 
Direct  R&D  Budget  Provided  by  State 
Government  Agencies 
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Profile  Information  for  Single-Applicant 
Companies 


Figure  1.6:  Percentage  of  Company’s 

Direct  R&D  Budget  Provided  by  100  Number  of  atp  Applicants 

Venture  Capitaiists  ^ 


80 


Percent  Provided  by  Venture  Capitalists 


Figure  1.7:  Percentage  of  Company’s 

Direct  R&D  Budget  Provided  Through  100  Number  of  atp  Applicants 
R&D  Contracts  With  Other  Companies 
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Appendix  I 

Profile  Information  for  Sin^e-Applicant 
Companies 


Figure  1.8:  Percentage  of  Company’s 
Direct  R&D  Budget  Provided  From 
internal  Funding 
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Figure  1.9:  Percentage  of  Company’s 
Direct  R&D  Budget  Provided  by 
Industry  Groups  or  Trade  Associations 


100  Number  of  ATP  Applicants 
90 
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Profile  Information  for  Single-Applicant 
Companies 


Figure  1.10:  Percentage  of  Company’s 

Direct  R&D  Budget  Provided  by  Other  100  Number  of  atp  Applicants 
Sources 
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Appendix  n 


Survey  Questions  and  Response 

Frequencies  for  Winners  and  Near  Winners 
of  ATP  Awards 


United  States  General  Accounting  Office 

frAO  Survey  of  ATP  Award  Winners  (Completed 

Questionnaire  with  Frequencies) 

INTRODUCTION 

3.  In  what  year  did  this  company  begin  operations?  ■ 

The  U.S.  General  Accounting  Office  (GAO),  an  agency 

_ 1  1803  to  1899 

that  examines  issues  and  programs  for  Congress,  is 

_ l_  1900  to  1929 

evaluating  some  of  the  impacts  of  the  Advanced 

Technology  Program  (ATP). 

_ 1  1930  to  1952 

6  1953  to  1973 

If  your  proposal  was  submitted  by  a  joint  venture,  answer 

38  1974  to  1994 

in  terms  of  your  experience  with  the  joint  venture’s 

proposal^  rather  than  with  a  participating  company.  We 

24  Missing 

recognize  that  joint  ventures  may  separate  various  duties 

among  participating  companies,  but  please  answer  the 

questions  as  best  you  can. 

4.  In  what  year,  if  any,  did  the  company  first  generate 
sales? 

If  the  proposal  was  submitted  by  a  single  applicant. 

35  1985  or  earlier 

answer  in  terms  of  your  experience  with  the  company’s 

proposal.  By  the  company  we  mean  the  entire  company 

3  1986 

(for  small  businesses),  or  the  unit  or  division  of  the 

_ ^  1987 

company  that  submitted  the  ATP  proposal  (for  large 

businesses). 

_ ^  1988 

Note:  These  frequencies  were  tabulated  after 

_ 2_  1989 

excluding  eleven  respondents;  therefore,  some  of  these 

_ 1_  1990 

statistics  may  not  match  those  in  the  body  of  the 

5  1991 

report.  See  appendix  III  for  details. 

_ 1992 

SECTION  I:  COMPANY  BACKGROUND 

3  1993 

INFORMATION 

_ l_  1994 

1 .  How  many  ATP  award  winners? 

_ 6^  Have  not  had  any  sales 

89  Winner 

23  Missing 

2.  Indicate  type  of  application. 

23  Joint  venture 

86  Single  applicant 

1 
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Survey  Questions  and  Response 
Frequencies  for  Winners  and  Near  Winners 
of  ATP  Awards 


5.  How  many  full-time  equivalent  employees  did  your 
company  have  when  you  applied  to  ATP?  Include 
outsourced  as  well  as  permanent  employees. 

37  0  to  100 

_ £  101  to  200 

_ ^  201  to  300 

_ I  301  to  400 

_ 0^  401  to  500 

16  Over  500 
25  Missing 

6.  In  what  year  did  the  member  companies  of  the  joint 
venture  agree  to  pursue  the  proposal  described  in  the 
ATP  proposal? 

_ 6_  1990 

_ ^  1991 

_ ^  1992 

_ 3_  1993 

66  Missing 

7.  In  what  year  did  the  joint  venture  actually  begin 
working  on  the  project? 

_ ^  1985  or  earlier 

_ 0^  1986 

_ 0_  1987 

_ 0^  1988 

_ 0_  1989 

_ ^  1990 

_ ^  1991 

10  1992 

_ ^  1993 

_ 5^  1994 

_ 0^  1995 

_ 0_  Not  started  yet 

_ ^  Don’t  know 

66  Missing 


8.  Did  the  joint  venture  come  together  as  a  new  group  to 
pursue  the  project  described  in  the  ATP  proposal,  or 
was  the  joint  venture  already  together  working  on  it? 

18  Joint  venture  came  together  to  pursue  the  ATP 
project 

_ ^  Joint  venture  was  pursuing  ATP  project 

together,  before  ATP 

3  Joint  venture  was  pursuing  other  unrelated 
projects  together,  before  ATP 

66  Missing 

9.  What  percentage  of  the  company’s  DIRECT  R&D 
budget  was  provided  hy  federal  government  agencies 
?  Base  your  answers  on  the  last  fiscal  year 
completed  before  you  applied  to  ATP. 

26  None 
17  1  to  20% 

10  21  to  40% 

_ £  41  to  60% 

_ 1_  61  to  100% 

25  Missing 

10.  What  percenUge  of  the  company’s  DIRECT  R&D 
budget  was  provided  by  state  government  agencies'! 
Base  your  answers  on  the  last  fiscal  year  completed 
before  you  applied  to  ATP. 

57  None 

_ 6^  1  to  20% 

_ 1_  21  to  40% 

_ 0^  41  to  60% 

_ 0^  61  to  100% 

25  Missing 
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Survey  Questions  and  Response 
Frequencies  for  Winners  and  Near  Winners 
of  ATP  Awards 


1 1 .  What  percentage  of  the  company’s  DIRECT  R&D 
budget  was  provided  by  industry  groups  or  trade 
associations^  Base  your  answers  on  the  last  fiscal 
year  completed  before  you  applied  to  ATP. 

58  None 

_ ^  1  to  20% 

_ 0^  21  to  40% 

_ 1  41  to  60% 

_ 0^  61  to  100% 

25  Missing 

12.  What  percentage  of  the  company’s  DIRECT  R&D 
budget  was  provided  by  other  private  companies'! 
Base  your  answers  on  the  last  fiscal  year  completed 
before  you  applied  to  ATP. 

40  None 
13  1  to  20% 

_ ^  21  to  40% 

_ i  41  to  60% 

61  to  100% 

25  Missing 

13.  What  percentage  of  the  company’s  DIRECT  R&D 
budget  was  provided  by  venture  capitalists?  Base 
your  answers  on  the  last  fiscal  year  completed  before 
you  applied  to  ATP. 

49  None 

_ ^  1  to  20% 

_ l_  21  to  40% 

_ 1^41  to  60% 

10  61  to  100% 

25  Missing 


14.  What  percentage  of  the  company’s  DIRECT  R&D 
budget  was  provided  by  internal  funding  from 
company  income?  Base  your  answers  on  the  last 
fiscal  year  completed  before  you  applied  to  ATP. 

20  None 
10  1  to  20% 

_ i  21  to  40% 

_ 41  to  60% 

28  61  to  100% 

25  Missing 

15.  What  other  funding  sources  did  you  use?  Base  your 
answers  on  the  last  fiscal  year  completed  before  you 
applied  to  ATP. 

_ ^  Banks 

_ L  Corporate  investors,  not  venture  capitalists 

_ ^  High  net  worth  individual 

_ L  internal  funding  from  interest,  dividends,  gifts, 

etc. 

_ l_  Licenses  related  to  R&D  contracts 

_ Owner’s  funds 

_ 2  Private  individuals,  not  venture  capitalists 

_ ^  Universities 

80  Missing 

SECTION  II;  ATP  PROPOSAL  HISTORY 

16.  Did  you  seek  funding  from  other  sources  to  pursue 
the  project  described  in  the  ATP  proposal  BEFORE 
you  sought  funding  from  ATP? 

30  Yes 
58  No 

_ Don’t  know 

_ 0^  Missing 
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Survey  Questions  and  Response 
Frequencies  for  Winners  and  Near  Winners 
of  ATP  Awards 


17.  How  long  did  you  seek  this  funding? 

13  Oto  12  months 

11  13  to  24  months 
_ £  Over  25  months 

61  Missing 

1 8.  How  many  full-time  equivalent  persons  were 
assigned  to  help  seek  funding  for  the  project  during 
this  period?  Include  outsourced  as  well  as  permanent 
employees. 

28  Oto 5 
_ 1^  Over  5 

60  Missing 

19.  Where  did  these  persons  seek  funding? 

12  U.S.  Federal  government  agencies  other  than 
ATP 

3  State  government  agencies 

10  Industry  groups/tradc  associations 

14  R&D  contracts  with  other  private  companies 
_ ^  Venture  capitalists 

9  Internal  funding  from  company  income 
_ ^  Other 

59  Missing 

20.  How  many  attempts  did  you  make  to  obtain  funding 
from  federal  government  agencies^ 

_ ^  None 

_ 5^  1  to  2  attempts 

_ £  3  to  4  attempts 

2  Over  4  attempts 

77  Missing 


21.  How  many  attempts  did  you  make  to  obtain  funding 
from  state  government  agencies*! 

_ 0^  None 

_ ^  1  to  2  attempts 

_ 0^  3  to  4  attempts 

_ 1_  Over  4  attempts 

86  Missing 

22.  How  many  attempts  did  you  make  to  obtain  funding 
from  industry  groupsitrade  associations*! 

_ ^  None 

_ £  1  to  2  attempts 

_ ^  3  to  4  attempts 

_ ^  Over  4  attempts 

79  Missing 

23.  How  many  attempts  did  you  make  to  obtain  funding 
from  other  private  companies*! 

_ 0_  None 

_ ^  1  to  2  attempts 

_ 0_  3  to  4  attempts 

10  Over  4  attempts 
75  Missing 

24.  How  many  attempts  did  you  make  to  obtain  funding 
from  venture  capitalists? 

_ ^  None 

_ ^  1  to  2  attempts 

_ ^  3  to  4  attempts 

_ £  Over  4  attempts 

80  Missing 
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Survey  Questions  and  Response 
Frequencies  for  Winners  and  Near  Winners 
of  ATP  Awards 


25.  How  many  attempts  did  you  make  to  obtain  funding 
from  internal  funding  from  company  income"? 

_ ^  None 

_ ^  1  to  2  attempts 

_ ^  3  to  4  attempts 

_ £  Over  4  attempts 

80  Missing 

26.  Were  any  of  your  attempts  to  obtain  funding 
unsuccessful  because  you  turned  down  funding  that 
had  terms  or  conditions  you  would  not  accept? 

_ I  Yes 

26  No 

_ ^  Don’t  know 

59  Missing 

27.  When  you  submitted  the  ATP  proposal*  did  you 
intend  to  pursue  the  project  whether  or  not  you 
received  ATP  funding? 

27  Yes 

_ ^  Probably  yes 

14  Uncertain 
17  Probably  no 
22  No 

28.  Did  you  intend  to  pursue  the  project  on  the  same 
schedule  as  described  in  the  ATP  proposal  or  on  a 
modified  schedule?  If  modified,  when  would  the 
milestones  be  met? 

_ 0^  Sooner  than  in  the  ATP  proposal 

_ ^  At  the  same  time  as  in  the  ATP  proposal 

34  Later  than  in  the  ATP  proposal 
53  Missing 


29.  Was  the  project  described  in  the  ATP  proposal 
underway  PRIOR  to  submission  of  the  proposal  to 
ATP? 

21  Yes 
68  No 

30.  Was  the  funding  level  of  the  on-going  project  less 
than*  about  the  same  as,  or  higher  than  the  amount 
requested  by  ATP? 

19  Less  than  the  ATP  request 

_ ^  About  the  same  as  the  ATP  request 

_ Higher  than  the  ATP  request 

68  Missing 

SECTION  HI.  PROJECT  STATUS  &  PROJECT 

RESULTS 

31 .  What  is  the  status  of  the  project  described  in  the  ATP 
proposal? 

55  Underway 
28  Completed 

_ ^  Discontinued 

_ ^  Other 

32.  How  satisfied,  if  at  all*  are  you  with  the  technical 
direction  and  progress/outcome  of  the  project? 

60  Very  satisfied 
24  Generally  satisfied 

_ 0^  Neither  satisfied  or  dissatisfied 

_ 3^  Generally  dissatisfied 

_ 0^  Very  dissatisfied 

4  Don’t  know 
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Survey  Questions  and  Response 
Frequencies  for  Winners  and  Near  Winners 
of  ATP  Awards 


33.  Which  statement  BEST  describes  the  results  you 
expected  to  have  at  the  end  of  the  project  at  the  time 
you  applied  to  ATP? 

55  Project  sold  commercially 
21  Process  used  internally 

_ ^  Product  or  process  used  by  another  firm  with 

compensation 

_ ^  Product  or  process  used  by  another  firm 

without  compensation 

_ ^  Don't  know 

34.  When  you  applied  to  ATP,  were  you  aware  of  other 
U.S.  companies  that  were  also  conducting  technical 
work  toward  objectives  of  the  ATP  proposaf? 

29  Yes 
59  No 

_ 3^  Don't  know 

35.  If  you  have  any  additional  comments  or  information 
you  would  like  to  provide  please  do  so  in  the  space 
below. 


Thank  yoUf  this  concludes  the  questionnaire! 
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Survey  Questions  and  Response 
Frequencies  for  Winners  and  Near  Winners 
of  ATP  Awards 


GAO 


United  States  General  Accounting  Office  _ _ _ 

Survey  of  ATP  Award  Near  Winners 
(Completed  Questionnaire  with  Frequencies) 


INTRODUCTION 

The  U.S.  General  Accounting  Office  (GAO),  an  agency 
that  examines  issues  and  programs  for  Congress,  is 
evaluating  some  of  the  impacts  of  the  Advanced 
Technology  Program  (ATP). 

If  your  proposal  was  submitted  by  a  joint  venture,  answer 
in  terms  of  your  experience  with  the  joint  venture’s 
proposal,  rather  than  with  a  participating  company.  We 
recognize  that  joint  ventures  may  separate  various  duties 
among  participating  companies,  but  please  answer  the 
questions  as  best  you  can. 

If  the  proposal  was  submitted  by  a  single  applicant, 
answer  in  terms  of  your  experience  with  the  company’s 
proposal.  By  the  company  we  mean  the  entire  company 
(for  small  businesses),  or  the  unit  or  division  of  the 
company  that  submitted  the  ATP  proposal  (for  large 
businesses). 

Note:  These  frequencies  were  tabulated  after 
excluding  eleven  respondents;  therefore,  some  of  these 
statistics  may  not  match  those  in  the  body  of  the 
report.  See  appendix  III  for  details. 

SECTION  I:  COMPANY  BACKGROUND 
INFORMATION 

1 .  How  many  ATP  award  near  winners? 

28  Near  winners 


3.  In  what  year  did  this  company  begin  operations? 

_ ^  1803  to  1899 

_ 3^  1900  to  1929 

_ 1930  to  1952 

_ ^  1953  to  1973 

_ 1  1974  to  1994 

_ ^  Missing 

4.  In  what  year,  if  any,  did  the  company  first  generate 
sales? 

12  1985  or  earlier 

_ 0^  1986 

_ 0_  1987 

_ 1_*1988 

_ 0^  1989 

_ 0^  1990 

_ 0^  1991 

_ ^  1992 

_ 0  1993 

_ 2^  1994 

_ 1995 

_ ^  Have  not  had  any  sales 

_ ^  Missing 


2.  Indicate  type  of  application 

_ ^  Joint  venture 

19  Single  applicant 
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Survey  Questions  and  Response 
Frequencies  for  Winners  and  Near  Winners 
of  ATP  Awards 


5.  How  many  full-time  equivalent  employees  did  your 
company  have  when  you  applied  to  ATP?  Include 
outsourced  as  well  as  permanent  employees. 

_ ^  0  to  100 

_ 0^  101  to  200 

_ ^  201  to  300 

_ 0^  301  to400 

_ ^  401  to  500 

_ Over  500 

_ ^  Missing 

6.  In  what  year  did  the  member  companies  of  the  joint 
venture  agree  to  pursue  the  proposal  described  in  the 
ATP  proposal? 

_ 2_  1989 

_ ^  1990 

_ ^  1991 

_ ^  1992 

_ ^  1993 

20  Missing 

7.  In  what  year  did  the  joint  venture  actually  begin 
working  on  the  project? 

_ 1_  1987 

_ l_  1990 

_ 1_  1991 

_ ^  1992 

_ ^  Not  started  yet 

_ _ ^  Don’t  know 

19  Missing 


8.  Did  the  joint  venture  come  together  as  a  new  group  to 
pursue  the  project  described  in  the  ATP  proposal,  or 
was  the  joint  venture  already  together  working  on  it? 

_ ^  Joint  venture  came  together  to  pursue  the  ATP 

project 

_ ^  Joint  venture  was  pursuing  ATP  project 

together,  before  ATP 

_ ^  Joint  venture  was  pursuing  other  unrelated 

projects  together,  before  ATP 

19  Missing 

9.  What  percentage  of  the  company’s  DIRECT  R&D 
budget  was  provided  by  federal  government  agencies 
?  Base  your  answers  on  the  last  fiscal  year 
completed  before  you  applied  to  ATP. 

_ 7_  None 

_ 5^  1  to  20% 

_ ^  21  to  40% 

_ I  41  to  60% 

_ 3_  61  to  100% 

_ ^  Missing 

10.  What  percentage  of  the  company’s  DIRECT  R&D 
budget  was  provided  by  state  government  agencies'} 
Base  your  answers  on  the  last  fiscal  year  completed 
before  you  applied  to  ATP. 

17  None 

_ 0  1  to  20% 

_ ^  21  to  40% 

_ 0^  41  to  60% 

_ 0^  61  to  100% 

_ ^  Missing 
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1 1 .  What  percentage  of  the  company’s  DIRECT  R&D 
budget  was  provided  by  industry  groups  or  trade 
associations^  Base  your  answers  on  the  last  fiscal 
year  completed  before  you  applied  to  ATP. 

19  None 

_ 0^  lto20% 

_ ^  21  to  40% 

_ ^  41to60% 

_ 0^  61  to  100% 

_ ^  Missing 

12.  What  percentage  of  the  company’s  DIRECT  R&D 
budget  was  provided  by  other  private  companies? 
Base  your  answers  on  the  last  fiscal  year  completed 
before  you  applied  to  ATP. 

13  None 

_ ^  1  to  20% 

_ 21  to  40% 

_ ^  41  to  60% 

_ l_  61  to  100% 

_ ^  Missing 

13.  What  percentage  of  the  company’s  DIRECT  R&D 
budget  was  provided  by  venture  capitalists?  Base 
your  answers  on  the  last  fiscal  year  completed  before 
you  applied  to  ATP. 

16  None 

_ 1  to  20% 

_ 0_  21  to  40% 

_ ^  41  to  60% 

_ 2_  61  to  100% 

_ ^  Missing 


14.  What  percentage  of  the  company’s  DIRECT  R&D 
budget  was  provided  by  internal  funding  from 
company?  Base  your  answers  on  the  last  fiscal  year 
completed  before  you  applied  to  ATP. 

_ 6^  None 

_ i  lto20% 

_ ^  21  to  40% 

_ 1_  41  to  60% 

_ ^61  to  100% 

_ ^  Missing 

15.  What  other  funding  sources  did  you  use?  Base  your 
answers  on  the  last  fiscal  year  completed  before  you 
applied  to  ATP. 

_ High  net  worth  individual 

27  Missing 

SECTION  II:  ATP  PROPOSAL  HISTORY 

16.  What  reasons,  if  any,  did  ATP  cite  for  declining  your 
ATP  award? 

15  Insufficient  business  plan 

_ 0^  Technology  not  precompetitive/generic 

_ 0^  Technology  posed  too  little  risk 

_ ^  Lack  of  available  ATP  funding 

14  Other 
0  Don’t  know 
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17.  What  were  the  “other”  reasons  cited  by  ATP? 

_ ^  ATP  wants  to  support  companies  not 

universities  in  technology  development 
.  proposals 

_ ^  Felt  company  did  not  have  an  adequate 

understanding  of  the  chemistry  of  why  the 
technology 

_ ^  Joint  venture  partners  disagreed  on  direction 

of  project  at  last  minute 

_ ^  Lack  of  an  adequate  transfer  plan 

_ ^  Lack  of  experience  of  team 

_ Limited  experience  in  commercialization 

_ ^  Matching  funds  on  low  side 

_ l_  Not  a  viable  invention 

_ ^  One  member  of  joint  venture  withdrew 

_ ^  Some  minor  technical  issues 

_ ^  Technical  content  was  incomplete 

_ ^  Too  little  company  commitment  for  money 

_ 1^  Uncertain  about  meeting  proposal  objectives 

18.  Did  you  seek  funding  from  other  sources  to  pursue 
the  project  described  in  the  ATP  proposal  BEFORE 
you  sought  funding  from  ATP? 

13  Yes 

15  No 

19.  How  long  did  you  seek  this  funding? 

_ ^  0  to  12  months 

_ ^  13  to  24  months 

_ ^  Over  25  nnonths 

16  Missing 

20.  How  many  full-time  equivalent  persons  were 
assigned  to  help  seek  funding  for  the  project  during 
this  period?  Include  outsourced  as  well  as  permanent 
employees. 

13  0to5 

_ 0_  Over  5 

15  Missing 


21 .  Where  did  these  persons  seek  funding? 

_ 1_  U.S.  Federal  government  agencies  other  than 

ATP 

_ ^  State  government  agencies 

_ 1_  Industry  groups/trade  associations 

_ 8^  R&D  contracts  with  other  private  companies 

_ ^  Venture  capitalists 

_ 5_  Internal  funding  from  company  inconnie 

_ ^  Other 

15  Missing 

22.  How  many  attempts  did  these  persons  make  to  obtain 
funding  from  federal  government  agencies? 

_ ^  None 

_ ^  1  to  2  attempts 

_ ^  3  to  4  attempts 

_ ^  Over  4  attempts 

22  Missing 

23.  How  many  attempts  did  these  persons  make  to  obtain 
funding  from  state  government  agencies? 

_ 0^  None 

_ 1_  1  to  2  attempts 

_ ^  3  to  4  attempts 

_ ^  Over  4  attempts 

26  Missing 

24.  How  many  attempts  did  these  persons  make  to  obtain 
funding  from  industry  groups/trade  associations? 

_ ^  None 

_ 0^  1  to  2  attempts 

_ 3  to  4  attempts 

_ 0^  Over  4  attempts 

27  Missing 
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25.  How  many  attempts  did  these  persons  make  to  obtain 

29.  When  you  submitted  the  ATP  proposal,  did  you 

funding  from  other  private  companies"} 

intend  to  pursue  the  project  whether  or  not  you 

received  ATP  funding? 

0  None 

— 

13  Yes 

2  1  to  2  attempts 

1  3  to  4  attempts 

_ 3^  Probably  yes 

5  Over  4  attempts 

_ Uncertain 

2  0  Missing 

_ ^  Probably  no 

_ 8_No 

26.  How  many  attempts  did  these  persons  make  to  obtain 

funding  from  venture  capitalists"} 

30.  Did  you  intend  to  pursue  the  project  on  the  same 

0  None 

schedule  as  described  in  the  ATP  proposal  or  on  a 
modified  schedule?  If  modified,  when  would  the 

_ 1^  1  to  2  attempts 

milestones  be  met? 

_ ^  3  to  4  attempts 

0  Sooner  than  in  the  ATP  proposal 

_ 1_  Over  4  attempts 

_ 0^  At  the  same  time  as  in  the  ATP  proposal 

24  Missing 

13  Later  than  in  the  ATP  proposal 

27.  How  many  attempts  did  these  persons  make  to  obtain 

_ Don’t  know 

14  Missing 

funding  from  internal  funding  from  company 

income"} 

0  None 

31 .  Was  the  project  described  in  the  ATP  proposal 
underway  PRIOR  to  submission  of  the  proposal  to 

_ ^  1  to  2  attempts 

ATP? 

_ 0^  3  to  4  attempts 

14  Yes 

_ 0_  Over  4  attempts 

13  No 

24  Missing 

1  Don’t  know 

28.  Were  any  of  your  attempts  to  obtain  funding 

32.  Was  the  funding  level  of  the  on-going  project  less 

unsuccessful  because  you  turned  down  funding  that 

than,  about  the  same  as,  or  higher  than  the  amount 

had  terms  or  conditions  you  would  not  accept? 

requested  by  ATP? 

_ 1  Yes 

11  Less  than  the  ATP  request 

10  No 

2  About  the  same  as  the  ATP  request 

15  Missing 

_ 1_  Higher  than  the  ATP  request 

14  Missing 

5 
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33.  What  percentage  of  the  funding  for  the  ongoing 
project  was  provided  by  federal  government  agencies 
7 

_ ^  None 

_ 0^  1  to  20% 

_ 1_  21  to  40% 

_ 41  to  60% 

_ ^  61  to  100% 

14  Missing 

34.  What  percentage  of  the  funding  for  the  ongoing 
project  was  provided  by  state  government  agenciesl 

10  None 

_ l_  1  to  20% 

_ 1  21  to  40% 

_ 0^  41  to  60% 

_ 0^  61  to  100% 

14  Missing 

35.  What  percentage  of  the  funding  for  the  ongoing 
project  was  provided  by  industry  groups  or  trade 
associations! 

14  None 

_ 0^  1  to  20% 

_ ^  21  to  40% 

_ 0^41to60% 

_ 0^  61  to  100% 

14  Missing 

36.  What  percentage  of  the  funding  for  the  ongoing 
project  was  provided  by  other  private  companies! 

12  None 

_ 1^  1  to  20% 

_ 0^  21  to  40% 

_ 0^  41to60% 

_ l_  61  to  100% 

14  Missing 


37.  What  percentage  of  the  funding  for  the  ongoing 
project  was  provided  by  venture  capitalists! 

12  None 

_ ^  I  to  20% 

_ 1^  21  to  40% 

_ 0^  41  to  60% 

_ 61  to  100% 

14  Missing 

38.  What  percentage  of  the  funding  for  the  ongoing 
project  was  provided  by  internal  funding  from 
company  income! 

_ ^  None 

_ ^  1  to  20% 

_ ^  21  to  40% 

_ 0^  41  to  60% 

_ ±  61  to  100% 

14  Missing 

SECTION  III.  PROJECT  STATUS  &  PROJECT 

RESULTS 

39.  We  understand  that  you  were  declined  an  ATP 
award.  Since  then,  have  you  begun  the  project 
described  in  the  ATP  proposal? 

14  Yes 

14  No 

40.  Is  the  project’s  schedule  the  same  as  described  in  the 
ATP  proposal,  or  is  it  modified?  If  modified,  when 
would  the  milestones  be  met? 

_ 0^  Sooner  than  in  the  ATP  proposal 

_ 3^  At  the  same  time  as  in  the  ATP  proposal 

13  Later  than  in  the  ATP  proposal 

■14  Missing 
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41.  What  percentage  of  the  funding  for  the  project 
described  in  the  ATP  proposal,  on  which  you  are  now 
working  is  being  provided  by  federal  government 
agencies^ 

_ ^  None 

_ 0^  1  to  20% 

_ 0^  21  to  40% 

_ 1^41  to  60% 

_ £  61  to  100% 

14  Missing 

42.  What  percentage  of  the  funding  for  the  project 
described  in  the  ATP  proposal,  on  which  you  are  now 
working  is  being  provided  by  subsequent 
competition  round  held  by  A  TFl 

_ ^  None 

_ 0^  1  to  20% 

_ 0_  21  to  40% 

_ 0^  41  to  60% 

_ 0^  61  to  100% 

14  Missing 

43.  What  percentage  of  the  funding  for  the  project 
described  in  the  ATP  proposal,  on  which  you  are  now 
working  is  being  provided  by  state  government 
agencies? 

11  None 

_ 3_  1  to  20% 

_ ^  21  to  40% 

_ 0^  41  to  60% 

_ 0^  61  to  100% 

14  Missing 


44.  What  percentage  of  the  funding  for  the  project 
described  in  the  ATP  proposal,  on  which  you  are  now 
working  is  being  provided  by  industry  groups  or 
trade  associations? 

11  None 

_ 1  to  20% 

_ 1_  21  to  40% 

_ 0^  41  to  60% 

_ ^61  to  100% 

14  Missing 

45.  What  percentage  of  the  funding  for  the  project 
described  in  the  ATP  proposal,  on  which  you  are  now 
working  is  being  provided  by  other  private 
companies? 

12  None 

_ l_  1  to  20% 

_ 0^  21  to  40% 

_ 0^  41  to  60% 

_ ^  61  to  100% 

14  Missing 

46.  What  percentage  of  the  funding  for  the  project 
described  in  the  ATP  proposal,  on  which  you  are  now 
working  is  being  provided  by  venture  capitalists? 

12  None 

_ 0_  1  to  20% 

_ l_  21  to  40% 

_ 0^41  to  60% 

_ 1_  61  to  100% 

14  Missing 
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47.  What  percentage  of  the  funding  for  the  project 
described  in  the  ATP  proposal,  on  which  you  arc  now 
working  is  being  provided  by  internal  funding  from 
company  income'? 

_ ^  None 

_ 0^  lto20% 

_ 2_  21  to  40% 

_ ^  41to60% 

_ £  61  to  100% 

14  Missing 

48.  How  long  did  you  look  for  funding  after  the  ATP 
proposal  was  declined?  If  you  looked  for  funding 
while  the  ATP  proposal  was  under  review,  also 
include  that  time  in  your  answer. 

_ ^  Did  not  look  for  funding  after  being  declined 

27  Missing 

49.  Where  did  you  look  for  funding  to  continue  or  begin 
the  project  described  in  the  proposal  before  you 
found  funding? 

_ 7  U.S.  Federal  government  agencies  other  than 

ATP 

_ 5^  Subsequent  competition  round  held  by  ATP 

_ 3^  State  government  agencies 

_ ^  Industry  groups/trade  associations 

_ ^  R&D  contracts  with  other  private  companies 

_ ^  Venture  capitalists 

_ ^  Internal  funding  from  company  income 

_ 0^  Other 

14  Missing 

50.  How  many  attempts  did  you  make  to  obtain  funding 
from  federal  government  agencies'? 

_ ^  None 

_ £  1  to  2  attempts 

_ 3^  3  to  4  attempts 

_ ^  Over  4  attempts 

21  Missing 


51 .  How  many  attempts  did  you  make  to  obtain  funding 
from  subsequent  competition  round  held  by  ATP'? 

_ 0^  None 

_ ^  1  to  2  attempts 

_ 3_  3  to  4  attempts 

_ ^  Over  4  attempts 

23  Missing 

52.  How  many  attempts  did  you  make  to  obtain  funding 
from  state  government  agencies'? 

_ 0_  None 

_ 3^  1  to  2  attempts 

_ 0^  3  to  4  attempts 

_ 0^  Over  4  attempts 

27  Missing 

53.  How  many  attempts  did  you  make  to  obtain  funding 
from  industry  groupsitrade  associations'? 

_ 0^  None 

_ ^  1  to  2  attempts 

_ ^  3  to  4  attempts 

_ ^  Over  4  attempts 

26  Missing 

54.  How  many  attempts  did  you  make  to  obtain  funding 
from  other  private  companies'? 

_ ^  None 

_ ^  1  to  2  attempts 

_ 0^  3  to  4  attempts 

_ 1  Over  4  attempts 

25  Missing 
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55.  How  many  attempts  did  you  make  to  obtain  funding  59.  How  satisfied,  if  at  all,  are  you  with  the  technical 
from  venture  capitalists'}  direction  and  progress/outcome  of  the  project? 


_ 0^  None 

_ 0^  1  to  2  attempts 

_ ^  3  to  4  attempts 

_ 0^  Over  4  attempts 

25  Missing 

56.  How  many  attempts  did  you  make  to  obtain  funding 
from  internal  funding  from  company  income'} 

_ 0^  None 

_ 1  to  2  attempts 

_ 0^  3  to  4  attempts 

_ 0^  Over  4  attempts 

27  Missing 

57.  Were  any  of  your  attempts  to  obtain  funding 
unsuccessful  because  you  turned  down  funding  that 
had  terms  or  conditions  you  would  not  accept? 

_ 0_  Yes 

10  No 

18  Missing 

58.  What  is  the  status  of  the  project  described  in  the  ATP 
proposal? 

11  Underway 

_ 0_  Completed 

_ ^  Discontinued 

_ ^  Other 

14  Missing 


_ 4^  Very  satisfied 

_ Generally  satisfied 

_ 0^  Neither  satisfied  or  dissatisfied 

_ ^  Generally  dissatisfied 

_ 0^  Very  dissatisfied 

14  Missing 

60.  Which  statement  BEST  describes  the  results  you 
expected  to  have  at  the  end  of  the  project  at  the  time 
you  applied  to  ATP? 

_ 6^  Project  sold  commercially 

_ £  Process  used  internally 

_ ^  Product  or  process  used  by  another  firm  with 

compensation 

_ 0_  Product  or  process  used  by  another  firm 

without  compensation 

_ Don’t  know 

14  Missing 

61.  When  you  applied  to  ATP,  were  you  aware  of  other 
U.S.  companies  that  were  also  conducting  technical 
work  toward  objectives  of  the  ATP  proposal? 

_ 6  Yes 

_ _ 8_  No 

14  Missing 

62.  Have  you  attempted  to  obtain  funding  from  sources 
other  than  ATP,  or  from  ATP  in  a  subsequent 
competition  round,  to  CONTINUE/BEGIN  the 
project  described  in  the  ATP  proposal? 

_ 9^  Yes 

_ ^  No 

_ ^  Don’t  know 

14  Missing 
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63.  Where  did  you  look  for  funding  to  continue  or  begin 
the  project  described  in  the  proposal  before  you 
found  funding? 

_ ^  U.S.  Federal  government  agencies  other  than 

ATP 

_ ^  Subsequent  competition  round  held  by  ATP 

_ 0^  State  government  agencies 

_ ^  Industry  groups/trade  associations 

_ ^  R&D  contracts  with  other  private  companies 

_ 0^  Venture  capitalists 

_ 0^  Internal  funding  from  company  income 

_ 0^  Other 

19  Missing 

64.  How  many  attempts  did  you  make  to  obtain  funding 
from  federal  government  agencies! 

_ 0_  None 

_ 3^  1  to  2  attempts 

_ 3_  3  to  4  attempts 

_ 0^  Over  4  attempts 

26  Missing 

65.  How  many  attempts  did  you  make  to  obtain  funding 
from  subsequent  competition  round  held  by  ATP! 

_ 0^  None 

_ ^  I  to  2  attempts 

_ 0^  3  to  4  attempts 

_ 0^  Over  4  attempts 

20  Missing 

66.  How  many  attempts  did  you  make  to  obtain  funding 
from  state  government  agencies! 

_ ^  None 

_ 0^  1  to  2  attempts 

_ ^  3  to  4  attempts 

_ 0^  Over  4  attempts 

28  Missing 


67.  How  many  attempts  did  you  make  to  obtain  funding 
from  industry  groupsUrade  associations! 

_ 0_  None 

_ ^  1  to  2  attempts 

_ 0^  3  to  4  attempts 

_ 0^  Over  4  attempts 

28  Missing 

68.  How  many  attempts  did  you  make  to  obtain  funding 
from  other  private  companies! 

_ 0_  None 

_ l_  1  to  2  attempts 

_ 0^  3  to  4  attempts 

_ l_  Over  4  attempts 

26  Missing 

69.  How  many  attempts  did  you  make  to  obtain  funding 
from  venture  capitalists! 

_ 0_  None 

_ ^  1  to  2  attempts 

_ 0^  3  to  4  attempts 

_ 0^  Over  4  attempts 

28  Missing 

70.  How  many  attempts  did  you  make  to  obtain  funding 
from  internal  funding  from  company  income! 

_ 0^  None 

_ 0^  1  to  2  attempts 

_ 0^  3  to  4  attempts 

_ 0^  Over  4  attempts 

28  Missing 
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71 .  Were  any  of  your  attempts  to  obtain  funding 

unsuccessful  because  you  turned  down  funding  that 
had  terms  or  conditions  you  would  not  accept? 

_ 0_  Yes 

_ l_  No 

19  Missing 


76.  If  you  have  any  additional  comments  or  information 
you  would  like  to  provide  please  do  so  in  the  space 
below. 


72.  Are  you  still  pursuing  funding  for  the  project? 

_ l_  Yes 

_ l_  No 

19  Missing 


Thank  you,  this  concludes  the  questionnaire! 


73.  How  long  have  you  been  seeking/did  you  seek 
funding  since  the  ATP  proposal  was  declined? 

_ £  0  to  12  months 

_ ^  13  to  24  months 

_ ^  Over  25  months 

19  Missing 


74.  How  many  full-time  equivalent  persons  were 
assigned  to  help  seek  finding  during  this  period? 
Include  outsourced  as  well  as  permanent  employees. 

_ 8^  0to5 

_ ^  Over  5 

19  Missing 


75.  When  you  applied  to  ATP»  were  you  aware  of  other 
U.S.  companies  that  were  also  conducting  technical 
work  toward  objectives  of  the  ATP  proposal? 

_ 7_ 

_ 6_  No 

_ ^  Don’t  know 

14  Missing 


11 
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Objectives,  Scope,  and  Methodology 


The  objective  of  this  report  was  to  examine  whether  funds  from  atp  are 
used  to  support  research  projects  that  would  not  have  been  funded  by  the 
private  sector,  or  if  it  replaces  private  funds  that  would  have  otherwise 
been  available.  To  meet  this  objective,  we  requested  from  atp  a  rank-order 
listing  of  all  the  applicants  that  received  a  score  from  the  Source 
Evaluation  Board*  for  their  atp  proposal  during  the  program’s  first  four 
rounds  of  competition  (1990-93). 

In  the  first  four  rounds  of  competition,  the  Source  Evaluation  Board  gave 
scores  only  to  proposals  that  had  been  determined  to  have  “very  high” 
scientific  and  technical  merit  and  that  had  passed  a  screening  stage  m 
which  it  was  determined  that  the  proposals  satisfied  the  program’s 
requirements.  The  Source  Evaluation  Board  then  assigned  a  score  to  all  of 
the  proposals  on  the  basis  of  a  business  review  and  all  the  criteria 
contained  in  Commerce’s  regulations.  According  to  atp  officials,  on  the 
basis  of  this  score,  the  proposals  with  strong  technical  and  business  merit 
were  ranked  and  recorded  on  a  list  before  the  final  oral  review  stage,  atp 
provided  that  list  to  us.  Using  this  list,  we  identified  those  that  received  atp 
awards  as  “winners”  and  those  that  did  not  receive  atp  funding  as  “near 
winners.” 

In  our  first  primaiy  research  question,  we  asked  the  near  winners  if  they 
had  continued  their  proposed  projects  using  other  funding  sources  after 
ATP  declined  to  fund  them.  We  developed  this  question  to  shed  light  on 
whether  atp  winners  (given  their  similarity  to  the  near  winners)  would 
have  continued  their  projects  using  other  funding  sources  if  atp  funding 
had  not  been  provided.  In  our  second  primary  research  question,  we  asked 
the  winners  and  near  winners  if  they  had  sought  funding  from  other 
sources  before  applying  to  atp.  This  question  provided  information  on 
whether  private-sector  sources  had  the  opportunity  to  fund  the  proposed 
projects  before  the  applicants  sought  federal  funding. 

Our  work  was  structured  in  three  phases.  First,  we  interviewed  atp 
officials,  winners,  and  near  winners  to  increase  our  understanding  of  atp’s 
review  process  and  the  applicants’  experiences  with  it.  We  spoke  with 
representatives  of  other  research  and  development  (r&d)  funding  sources, 
such  as  the  National  Venture  Capital  Association.  We  also  reviewed  the 


^For  each  competition,  ATP  forms  a  Source  Evaluation  Board  to  rank  the  proposals.  A  typical  board 
consists  of  about  a  dozen  senior-level  managers  from  the  National  Institute  of  Standards  and 
Technology  (NIST),  supplemented  with  additional  technical  consultants  from  NIST  and  other  federal 
laboratories.  The  board  members’  backgrounds  vary  widely.  The  board  may  include,  for  example,  an 
electrical  engineer,  a  chemist,  a  biotechnologist,  a  materials  scientist,  a  computer  scientist,  and  others 
with  business  and  economics  expertise. 
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relevant  economic,  policy,  and  evaluation  literature  and  consulted  with 
outside  experts  on  the  overall  design  of  our  assessment. 

Next,  we  designed  a  computer-aided  telephone  interview  in  which  we 
requested  several  pieces  of  information  from  each  applicant.^  Specifically, 
we  asked  questions  on  (1)  the  applicant’s  general  characteristics,  such  as 
the  size  of  the  company  and  its  sources  of  r&d  funding;  (2)  the  history  of 
the  project  put  forth  in  the  atp  proposal,  such  as  whether  the  project  was 
under  way  before  the  applicant  requested  funding  from  atp;  and  (3)  the 
project’s  status  (for  winners  and  near  winners  that  continued  their 
projects  using  other  funding  sources).  We  got  expert  review  of  a 
preliminary  version  of  the  questions  from  knowledgeable  consultants. 

To  test  the  validity  of  the  questions,  we  pretested  a  draft  survey 
instrument  with  three  atp  award  winners  and  four  near  winners.  We 
selected  them  using  the  following  factors:  the  round  of  competition  in 
which  the  application  was  submitted,  geographic  location,  type  of 
applicant  (single  company  applicant  or  joint  venture),  company’s  size, 
proposal  technology  area,  and  award  status  (award  winner  or  near 
winner).  We  conducted  the  first  three  pretests  in  person  in  Gaithersburg, 
Maiyland  and  Somerset,  New  Jersey;  we  conducted  the  remaining  four  by 
telephone  with  winners  and  near  winners  located  in  Ann  Arbor  and 
Auburn  HiUs,  Michigan,  and  San  Jose  and  Menlo  Park,  California.  On  the 
basis  of  the  conunents  and  reactions  firom  the  experts’  review  and  our 
pretests,  we  revised  the  telephone  interview  questions  so  that  they  would 
be  uniformly  interpreted  and  imderstood. 

In  the  final  phase  of  our  work,  we  conducted  telephone  interviews  with  all 
the  applicants  that  qualified  as  winners  or  near  winners  during  atp’s  first 
four  rounds  of  competition,  a  total  of  128  (89  winners  and  39  near 
winners).  Our  survey  achieved  a  100-percent  response  rate.  In  our  findings 
for  both  research  questions,  we  excluded  5  near  winners,  reducing  our 
total  number  of  respondents  to  123.  We  did  this  because  these  five  near 
winners  indicated  that  atp  had  disqualified  them  late  in  the  review  process 
because  new  information  indicated  that  their  proposals  did  not  satisfy  the 
program’s  basic  requirements.  For  example,  in  one  instance  atp  decided 
that  the  applicant  would  do  the  project  without  atp  funding  and  in  another 
ATP  decided  that  the  project  did  not  focus  on  precompetitive  or  generic 
research.  In  our  findings  for  the  second  research  question  only,  our  total 
of  near  winners  drops  to  28  because  6  near  winners  eventually  received 


computer  was  programmed  to  skip  questions  that  were  irrelevant  to  the  individual  respondent 
For  example,  those  involved  in  joint  ventures  were  not  asked  for  the  year  of  their  first  sale.  In  these 
cases,  the  computer  tabulated  the  results  as  a  missing  response.  See  app.  H. 
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ATP  funding  in  a  subsequent  round  of  competition,  eliminating  them  from 
our  consideration  because  we  focused  only  on  near  winners  that  found 
funding  from  sources  other  than  atp. 

To  examine  how  certain  characteristics  affect  whether  the  near  winners 
continued  their  projects,  we  calculated  “odds  ratios.”  Odds  ratios  measure 
the  association  between  two  variables.  The  closer  the  odds  ratio  to  1.00, 
the  weaker  the  association;  the  further  from  1.00,  the  stronger.^  To 
illustrate,  table  in.l  reports  the  niunber  of  joint-venture  and 
single-applicant  near  winners  that  did  and  did  not  continue  their  project 
after  atp  declined  to  fund  them. 


Table  III.1 :  Number  of  Joint-Venture 
and  Single-Applicant  Near  Winners 
That  Continued  Their  Projects  After 
ATP  Declined  to  Fund 


Joint  ventures 

Single  applicants 

Continued 

3 

11 

Did  not  continue 

6 

8 

The  odds  ratio  is  calculated  through  cross  multiplication  and 
division — (11x6)  divided  by  (3x8) — ^for  a  value  of  2.75.  Rotmding  to  3,  we 
interpret  this  odds  ratio  to  mean  that  single-applicant  near  winners  were 
about  three  times  more  likely  than  joint-venture  near  winners  to  continue 
projects  after  atp  declined  to  fund  them. 


®For  more  detail  on  the  theory  underlying  odds  ratios  and  their  calculation,  see  William  Page, 
“Interpretation  of  Goodman’s  Log-Linear  Model  Effects:  An  Odds  Ratio  Approach,"  Sociological 
Methods  &  Research,  Vol.  5,  No.  4,  May  1977. 
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Note:  GAO  comments 
supplementing  those  in  the 
report  text  appear  at  the 
end  of  this  appendix. 


/SS 


V 


THE  SECRETARY  OF  COMMERCE 

Washington,  D.C.  20230 


'*■  1 


Mr.  Victor  S.  Rezendes 
Director,  Energy  and  Science  Issues 
Resources,  Conravunity,  and  Econoinic 
Development  Division 
U.S.  General  Accounting  Offices 
Washington,  D.C,  20548 

Dear  Mr.  Rezendes: 

We  appreciate  the  opportunity  to  comment  on  the  draft 
report,  Performance  Measurement:  The  Advanced  Technology  Program 
and  Private  Sector  Funding,  (GAO/RCED-'96-47 ,  Code  307723).  We 
note  that  the  General  Accounting  Office  (GAO)  has  not  allowed  us 
the  usual  30  days  to  provide  comments  on  this  draft  report.  Our 
comments  are  enclosed. 

Some  of  the  concerns  noted  in  the  enclosure  are  sufficiently 
important  that  unless  they  are  taken  into  account  by  GAO  before 
the  report  is  finalized,  we  believe  the  report’s  conclusions 
would  be  potentially  misleading  at  best  and  erroneous  at  worst. 
Much  of  the  report  is  well-done;  and  so,  if  our  recommendations 
are  taken  into  account  conscientiously,  this  could  be  a  valuable 
and  generally  accurate  report. 

If  your  staff  requires  additional  information  about  the 
Department’s  response  to  your  report,  they  may  contact  the 
Advanced  Technology  Program’s  Senior  Economist,  Rosalie  Ruegg,  at 
(301)  975-6135. 

Sincerely, 

Ronald  H.  Brown 

Enclosure 
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December  14,  1995 


Department  of  Commerce  (DOC)  Comments  on  GAO  Draft  Report 
Measuring  Performance :  The  Advanced  Technology  Program 

Our  most  important  concern  is  that  in  stating  the  conclusion,  GAO 
must  not  bury  in  the  text  of  the  report  critical  caveats  that 
might  be  overlooked  by  the  casual  reader.  These  caveats  are 
critically  important  to  the  conclusions  that  readers  may  draw. 
With  these  caveats,  readers  will  correctly  interpret  the  GAO’s 
findings.  Without  them,  readers  will  likely  draw  invalid 
conclusions . 

The  GAO’s  survey  results  support  the  conclusion  that  the  ATP  is 
meeting  its  objective  of  funding  projects  that  either  would  not 
be  pursued  at  all  or  projects  that  would  have  been  pursued 
without  ATP  funding,  but  at  a  much  slower  pace. 

-  Of  the  respondents  who  said  they  intended  to  pursue  the 
project  whether  or  not  they  received  ATP  funding,  nearly  all 
indicated  that  without  the  ATP  award  their  milestones  would 
be  met  later  than  those  set  forth  in  the  proposal. 

-  The  remainder  of  those  who  responded  (other  than 
“uncertain)  indicated  that  they  would  have  been  unlikely  to 
pursue  the  research  at  all  without  the  ATP. 

-  The  fact  that  some  of  the  near  winners  were  able 
subsequently  to  obtain  funding  is  perfectly  consistent  with 
ATP’s  decision  not  to  fund  them.  If  we  believe  that  an 
applicant  does  not  need  ATP  funds  to  pursue  the  project,  we 
do  not  fund  them.  (One  of  our  selection  criteria  is  “Degree 
to  which  ATP  support  is  necessary.”  ) 

-  The  fact  that  the  scale  of  on-going  research  was  expanded 
by  the  ATP  award  is  consistent  with  ATP’s  goal. 

-  The  fact  that  a  higher  percentage  of  ATP  awardees 
indicated  satisfaction  with  the  technical  direction  and 
progress/ outcome  of  their  project  than  the  near  winners 
continuing  may  indicate  that  those  continuing  without  ATP 
were  not  able  to  pursue  the  full  project. 

-  As  GAO  correctly  concludes,  the  survey  results  indicate 
that  the  ATP  successfully  fosters  joint  research  ventures. 
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Because  it  decouples  the  results  of  two  appropriately  coupled 
questions,  the  GAO’s  first  statement  on  pages  3  and  11  is 
misleading,  and  needs  to  be  corrected. 

-  Our  experience  has  been  that  if  companies  are  asked, 

“Would  you  have  pursued  the  project  without  ATP  funding?” 
there  is  a  very  high  potential  for  multiple  interpretations 
of  the  question.  At  one  extreme,  some  companies  assume  they 
are  being  asked  whether  without  ATP  funds  they  would  have 
continued  any  amount  of  work  at  all  in  the  general  subject 
area  of  the  proposal.  Others  may  assume  they  are  being 
asked  whether  the  precise  project  would  have  been  pursued 
with  no  changes.  Still  others  will  take  an  interpretation 
somewhere  between  these  two  extremes.  For  this  reason,  we 
have  found  that  the  more  meaningful  question  to  ask  in  this 
regard  is, 

“Without  ATP  cost-sharing,  how  would  your  R&D  in  this 
specific  area  have  differed  with  regard  to  schedule, 
scope  of  project,  riskiness  of  technical  and  business 
goals,  collaborations,  etc,?” 

-  The  GAO  appropriately  asked  a  follow-on  question  about 
whether  the  project  schedule  would  have  been  different 
without  the  ATP,  but  unfortunately,  it  failed  to  include  the 
other  important  ways  a  project  can  be  altered  —  e-g»f 
scope,  scale,  and  riskiness  of  goals  —  in  the  follow-on 
question.  And,  according  to  two  other  third-party  surveys, 
if  given  the  chance,  the  companies  would  also  have  indicated 
a  smaller-sized  effort,  less  ambitious  research  goals, 
and/or  a  less  comprehensive  project  without  the  ATP  award. 

-  More  importantly,  the  GAO  failed  to  report  in  its  summary 
of  results  and  conclusions,  the  results  of  the  follow-on 
question  that  revealed  that  nearly  all  of  those  who  said 
they  planned  to  pursue  the  research  without  ATP,  said  that 
the  research  schedule  would  be  slower  without  ATP. 

(However,  the  GAO  did  correctly  report  in  part  on  the  bottom 
of  page  5  the  results  of  the  paired  questions,  and  that 
information  deserves  to  be  highlighted  more.) 

To  address  these  concerns,  we  strongly  urge  the  GAO  to  change  the 
first  paragraph  of  the  executive  summary  section  titled  “Results 
in  Brief”  on  page  3  and  the  “Conclusions”  section  on  page  11  to 
read  along  the  lines  of  the  following.  (This  wording  is  fully 
consistent  with  the  caveats  noted  on  page  5  of  the  GAO  report.) 

ATP  funds  research  projects  that  either  would  not  have  been 
funded  at  all  without  the  ATP,  or  would  likely  have  been 
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See  comment  4. 


funded  by  others  but  at  a  slower  pace  and/or  with  less 
ambitious  and  less  risky  goals.  Nearly  all  of  those  who 
said  they  would  have  pursued  their  project  to  some  extent 
without  ATP  cost  sharing  indicated  that  the  project  schedule 
would  have  been  modified  without  ATP  cost  sharing  and  that 
milestones  would  be  met  later  than  those  set  forth  in  the 
proposal  to  ATP.  The  survey  questions  did  not  ask  whether, 
in  such  cases,  the  technical  goals  would  also  have  been  less 
ambitious  and  less  risky  without  ATP  cost  sharing,  but  other 
surveys  carried  out  by  ATP  to  which  we  had  access  indicate 
that  this  also  might  be  so  in  many  cases. 

-  We  also  urge  that  the  report  state  clearly  that 
accelerating  the  pace  of  research  is  but  one  of  the  ways 
that  the  ATP  funding  may  alter  those  projects  that  might 
have  been  pursued  without  ATP. 

Were  these  recommended  changes  to  be  made,  we  believe  the  GAO 
survey  can  provide  useful  insight  into  the  differences  between 
the  winners  and  near  winners  of  ATP  awards. 

Paragraph  1  of  page  1  of  the  report  states  that  the  ATP  funding 
for  FY95  was  $431  million.  That  is  incorrect.  Because  of 
the  $90  million  rescission,  the  actual  funding  for  FY95  was 
$341  million. 
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GAO’s  Comments 


The  following  are  gao’s  comments  on  the  Department  of  Commerce’s 
letter  dated  December  21, 1995. 

1.  WhUe  our  draft  Conclusions  section  referred  to  the  pace  of  research 
projects,  we  have  added  a  sentence  to  our  report’s  Results-in-Brief  on  this 
point.  In  our  opinion,  however.  Commerce’s  oyeraU  conclusion  based  on 
our  survey  results  overlooks  a  number  of  significant  points.  First,  although 
ATP  appears  to  enable  applicants  to  complete  their  research  projects  faster, 
companies  still  find  it  worthwhile  to  pursue  the  projects,  although  on  a 
slower  schedule,  without  atp  funds.  Second,  when  asked,  “What  reasons,  if 
any,  did  atp  cite  for  declining  your  atp  award?,”  only  one  applicant  said 
that  ATP  had  decided  the  project  wovild  be  done  without  atp  funding. 
(Subsequently,  that  project  did  not  find  funding  elsewhere.)  Third,  while 
our  survey  results  do  indicate  that  a  higher  percentage  of  atp  awardees 
indicated  satisfaction  with  the  technical  direction  and  progress/outcome 
of  their  projects  than  the  near  winners,  nothing  in  our  survey  supports 
Commerce’s  conclusion  about  what  those  results  indicate. 

2.  Our  draft  Conclusions  section  referred  to  the  pace  of  continued 
research  projects,  and  we  have  added  a  sentence  to  our  Results-in-Brief 
section  on  this  point.  While  Commerce  is  concerned  with  multiple 
interpretations  of  one  of  our  survey  questions,  we  reduced  the  potential 
for  multiple  interpretations  by  instructing  our  interviewers  to  say 
specifically,  “I’ll  be  asking  you  several  questions  about  the  history  and 
status  of  ’the  project  described  in  your  atp  proposal,’  which  I’ll  sometimes 
refer  to  as  ’the  project.’  By  this  we  mean  a  project  that  you  consider  to  be 
essentially  the  same  as  the  one  in  the  proposal.”  If  further  clarification  was 
needed,  the  interviewer  would  add,  “To  be  ’essentially  the  same  project,’  it 
should  focus  on  the  same  technical  work  as  the  one  in  the  atp  proposal.” 

We  used  this  wording  to  allow  the  respondent  to  rely  on  his  or  her  own 
judgment  in  determining  if  the  work  that  had  continued  was  still  the  same 
project — despite  changes  in  scope,  schedule,  and  riskiness,  among  other 
things — or  if  in  the  respondent’s  judgment,  changes  have  resulted  in  a 
different  project  altogether.  At  a  minimum,  the  project  had  to  include  the 
same  technical  work,  even  though,  for  example,  some  intended 
commercial  applications  of  the  work  had  changed. 

3.  As  noted  in  comments  1  and  2,  we  feel  that  Commerce’s  conclusion 
overlooks  a  number  of  significant  points  based  on  our  survey  results. 

4.  We  have  made  the  suggested  change. 
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